The present invention relates to a method switching the video component(s) of a first digital, audio-visual program onto the video component(s) of a second digital, audio-visual program.
The invention applies to the field of digital, audio-visual services wherein digital programs consist of at least one video component and may comprise several video components, none or several audio components, and none or several data components. The invention relates only to the video components.
Pertinent digital audio-visual services include those using the so-called MPEG-2 standard (Moving Pictures Expert Group) ISO/IEC JTC1 IS 13818. This standard is described in several documents each corresponding to a distinct portion (video, audio, system, conformity etc.)
Nevertheless it is to be understood that the present invention is not restricted to solely following this particular MPEG-2 standard but also covers any digital, audio-visual service application.
Three kinds of pictures are defined generally, and in particular in the video portion of the MPEG-2 standard:
The so-called I images (Intra-coded pictures] that do not refer to any other picture; these are moderate compression-ratio pictures; their coding does not imply using methods of motion estimates,
The so-called P pictures (predictive code pictures) which are coded by using prediction methods estimating motion that are based on a preceding picture which may be a type I picture or a type P picture,
The so-called B pictures (bidirectionally predictive-coded pictures) which are coded using motion prediction methods based on a preceding picture (called back prediction for displays) which then are I or P type.
In general the video components discussed in the present invention therefore consist of an ordered string of Intra, predictive- or bidirectional pictures of which the order varies depending on involving the order of presentation or the order of transmission.
One illustrative application of a method of the present invention concerns chaining different TV programs. It involves switching from a first program selected from a given number of a first multiplex"" programs to a second program selected from a given number of a second multiplex"" programs and possibly, later, in switching again from the second program to the first.
In other words, a first program is disconnected in order to connect to second program, and back. These operations entail two multiplexes of the same syntax which are coded in real time or are retrieved from a storage system.
This application incurs a difficulty which shall now be described in relation to FIGS. 1 and 2 and within the narrower scope of a system adhering to the MPEG-2 standard. Before that, however, said standard will be elucidated.
The audio and video portions of the MPEG-2 standard describe how to digitally code the video and audio data. According to the system part of the MPEG-2 standard, the resultant compressed elementary streams are transformed into elementary stream packets PES (Packetized Elementary Streams). The PES packets containing video data may be of constant or variable sizes. The illustrative embodiment described below relates to using PES video packets of variable size, each PES packet comprising a single picture (possibly with sequence headers or picture groups GOP headers), the pictures being aligned at the beginning of the PES packet. The PES packets moving in the same elementary stream are identified by the same identifier called Stream ID.
The system standard MPEG-2 also specifies that the PES packets (video, audio or data) may be multiplexed for local use for instance with PES signaling packets. Such a multiplex may be stored on disk such as those known as DVD (Digital Versatile Disk), for instance it relates to the program of MPEG-2 syntax.
According to the system MPEG-2 standard, the elementary stream formed into PES packets may then be segmented into fixed-size packets called transport packets (TP). Such a fixed-size packet is identified by a packet identifier (PID). A stream of PES packets having the same Stream ID can only move in transport packets with the same packet identifier PID. The transport packets moving a particular elementary stream (and the PES packets having for the first time segmented this elementary stream) may be multiplexed with other TP transport packets moving other elementary streams, not excluding however TP transport packets containing signaling information etc. The TP transport multiplexes then will be suitable for transmission and their organization then will be determined by the transport MPEG-2 syntax.
The video MPEG-2 standard calls provides six hierarchical levels for the syntax of a video elementary stream: the sequence, the so-called group of pictures (GOP) which is the only optional level, the picture (type I, P or B), the image slice, the macro block and the block.
A video sequence begins with a sequence header which is followed by one or more sets of pictures. According to MPEG-2, these sets of pictures may be combined into groups of pictures GOP. One group of pictures GOP is characterized by a header of a group of pictures GOP situated ahead of the pictures. In the order of transmission, the group of pictures GOP begins with an I picture followed by a given number of bidirectional pictures B and predictive pictures P, and this in a rigorous order. A group of pictures GOP is said to be open when its first pictures B refer to the last picture P of the preceding set or to the group of pictures GOP; it is called closed when there is no such reference. The invention is appropriate for the video elementary streams codes as an open group of pictures GOP.
In general the program video components consists of strings of sequences with one group of pictures GOP per sequence. In a stretch of terminology, any group of pictures GOP shall be called hereafter such a group which is preceded by a sequence header.
Each picture comprises a header and data consisting of a number of slices. A slice pools several continuous macro-blocks in the picture. In conventional digital television (format 4; 2; 0), one micro-block is the combination of four luminance blocks, one block of a first chrominance component for the same pixels and one block of a second chrominance component also for the same pixels. Illustratively one block contains the data of eight by eight pixel squares.
The illustrative implementation of the invention described below is situated at the level of the MPEG-2 transport portion, however the operations carried out at the elementary level and at the level of the PES packets are directly applicable to the MPEG-2 program standard.
Even if all precautions have been taken to terminate the first video component at the sequence end (and at the end of group of pictures GOP) and to start the second video component at the beginning of the sequence (and hence at the beginning of the group of images GOP), there is a major problem when switching a first digital, audio-visual program of conventional television onto a second one, namely the degradation of the resultant picture. It is known that this degradation takes place when the coding of the first pictures B refers to the last picture P or I of the preceding sequence. If the concept of group of pictures GOP is used, this coding is called xe2x80x9ccoding in open group of images GOPxe2x80x9d: the appearance of a mosaic in the restored picture is caused by the rupture of the rear reference of the first bidirectional pictures B of each new group of pictures.
Be it borne in mind that each program may comprise several video components, for instance components which are of different views in the program (for instance from several cameras). The invention does apply to each of these components; however, for the sake of clarity, only one video component shall be discussed for each program.
Be it also noted that FIG. 1 is free of the PES packet and transport packet levels. This FIG. 1 represents an elementary video stream F1 of a first program and an elementary video stream F2 of a second program. Each stream F1, F2 consists of a string of groups of pictures GOP comprising pictures I, B and P in the order of transmission. The sequence headers are omitted even though each group of pictures GOP beginning with a picture 1 does comprise one. The headers of the groups of images GOP are also omitted.
The disconnection illustrated by the arrow DEC takes place at the end of a group of pictures GOP1 of the first program and at the beginning of a group of pictures GOP2 of the second program. Be it borne in mind that if the second multiplex is in registration, no problem shall arise to start at a group of pictures of the second program at the end of a group of picture of the first program. On the other hand and as is the case in FIG. 1, if the second multiplex is transmitted in real time, disconnection may require retarding the second program in order to constitute a delayed elementary video stream F2s. For that purpose it may be necessary to employ a buffer memory to store a set of pictures (and its packaging into PES packets and where called for into transport packets) of the elementary video stream of the second program. Frequently the delay introduced by said memory will be undesirable. However the means to avoid it are not considered in this invention.
Similarly reconnection denoted by the arrow RAC takes place at the end of a group of pictures GOP2xe2x80x2 of the elementary video stream of the second program to start again on a group of images GOP1xe2x80x2 of the elementary video stream of the first program. For that purpose and as was done in the disconnection case, a delay may be introduced in the elementary video stream of the first program. Consequently a buffer memory would be needed to store a group of pictures of the elementary video stream of the first program or using the same means as in disconnection to allow adding the delay caused by this memory.
The major problem introduced by the disconnection or reconnection of digital audio-visual programs in conventional television, and in general by any switching of a first digital audio-visual program onto a second one, is the degradation of the resultant picture. It is known that this degradation occurs when the group-of-picture coding is an open group of pictures GOP wherein the first bidirectional pictures B refer to the last predictive picture P or picture I of the preceding group of images GOP.
Furthermore this type of coding is being used increasingly because, for the sake of good compression, the commercial coders do not as a rule make use of closed groups-of-pictures coding.
The MPEG-2 standard allows denoting in the optional header of any group of pictures GOP that the first bidirectional picture(s) B of the group of pictures GOP cannot be appropriately decoded because the reference picture used for prediction no longer is valid. This is the broken-link indicator. On one hand this information only allows correcting the problem if the concept of group of pictures GOP is used and on the other hand this label is presently useless because in general neither the applications nor the MPEG-2 standard require processing by receivers.
The appearance of a mosaic in the restored picture is caused by the rupture of the rear reference of the first bidirectional pictures B of the new group of pictures (the group GOP2 of the second video elementary stream at disconnection or the group GOP1xe2x80x2 of the first video elementary stream at reconnection in FIG. 1).
FIG. 2 schematically shows the end of a group of pictures GOP1 of an elementary video stream of a first program and the beginning of a group of pictures GOP2 of an elementary video stream of a second program, the first program being switched onto the second. These groups are shown in the order of transmission of the pictures on the upper line and in the order of display of the pictures after decoding on the lower line. The arrow A represents the time of switching. Moreover FIG. 2 shows by means of arcuate arrows the references from one another or several other pictures. Illustratively the picture B2 refers to both the picture I1 and the picture P3 and this picture P3 refers to the picture I1. The numerals associated with the pictures correspond to their order of presentation (or display).
At the time of coding, the first bidirectional picture B0 of the group of pictures GOP2 exhibits back prediction with a reference on the picture I1 of the group of pictures GOP2 and a forward prediction with a reference on the last picture P11 of the preceding group of pictures GOP1. Following switching, this last reference turns out to be to the last picture P11 of the old group of pictures GOP1 of the first-program video component which does not relate to the same program. Accordingly the back prediction of the first picture B0 is not carried out on the good picture P and this circumstance entails poor decoding of this picture B0. In the embodiment of FIG. 2, this picture B0 shall be composed of references (and hence of parts) of the picture P11 of the old program and of references (and hence of parts) of the first picture I1 of the new program, whereby, on restoration, said xe2x80x9cmosaicxe2x80x9d materializes.
The objective of the present invention is a switching method applied a video component of a first, digital audio-visual program onto a video component of a second, digital audio-visual program, said method being such that the restoration shall be free of mosaics at the time of switching and that the sole discontinuity exhibited to the user shall be that of program change.
For that purpose a method of the invention to switch the video components of a first, digital audio-visual program onto the video component(s) of a second, digital audio-visual program is characterized in that it switches at the beginning of the first set of pictures comprising an Intra picture of the video component of the second program encountered after the switch command, and, in the order of picture transmission, in replacing each of the bidirectional pictures of this first set of pictures, which refers to pictures belonging to the set of pictures preceding it before switching, with a bidirectional picture of which the video coding is carried out independently of the coding contents of the replaced picture and of the contents of the pictures two which it refers.
In another feature of the invention, said vectors approximating the motion of the replacement picture, or each one, are set to a zero value.
In another feature of the invention, said replacement picture(s) relate uniquely to the Intra picture of the first set of pictures of the second-program video component encountered following the switch command. Said replacement picture(s) advantageous are frame pictures similar to digital television pictures and thus comprise a TOP and a BOTTOM frame and their predictions are field based predictions. The TOP frame and the BOTTOM frame of the replacement picture, or of each of them, then refer to the single TOP frame of the first Intra picture of the set of pictures of the second-program video component.
In another feature of the invention, said replacement picture or each of them refers uniquely to the last predicted picture or Intra picture preceding the switching of the second-program video component, as may be the case when the sets of pictures of the first and second are joined. In this case too the said replacement picture(s) advantageously shall be frame pictures comprising a TOP frame and a BOTTOM frame similar to digital television pictures and their predictions are field based type predictions. The TOP frame and the BOTTOM frame of the replacement picture, or of each of them, then shall refer to the single BOTTOM frame of the last predicted or Intra picture of the last set of components of the first-program video component.
Said pictures of each of said sets of pictures from the first and second programs are respectively carried by PES (packetized elementary streams) packets, each PES packet comprising a single picture and beginning with a video access unit (pictures"" beginning, beginning of sequences or of sets of pictures). In the invention, said method consists in:
determining the first PES packet of the second-program video component which is present following the switch command in the stream of PES packets and which comprises a sequence header in order to determine the beginning of the pictures of the second program to which the switching is applied,
beginning with this PES packet, a position is assumed on the first bidirectional picture B of this first set of pictures of the second program,
replacing the payload data with the replacement picture data in the above PES packet corresponding to a B-picture beginning,
once all the data of the replacement picture have been inserted, replacing the data of the payload of this PES packet with video stuffing such as xe2x80x9c0xe2x80x9d figures until reaching the last data octet of the PES packet,
if the number of bidirectional pictures preceding the first predicted picture in said set of pictures is larger than 1, repeating the two above stages for the ensuing pictures B, namely beginning with the next PES packet of this video component.
Said pictures of each of said sets of pictures from the first and second programs illustratively also may be moved by streams of transport packets, each transport packet being fitted with an PUSI indicator which when set at 1 denotes that said packet contains the beginning of a PES packet containing a single picture and beginning with a video access unit, where certain transport packets shall bear transport information such as a random access indicator RAI which, when set at 1, denotes that the next transport packet contains the first data of a sequence (and of a set of pictures). Said method then is characterized by consisting in:
determining the first transport packet TP of the second-program video component which is present following the switch command in the transport stream that comprises a random access indicator RAI set at 1 in order to determine the beginning of the first set of picture of the second program on which the switching is operative,
setting at 1 the discontinuity indicator for the adaptation field AF of this transport packet TP,
starting with this transport packet TP with random access indicator RAI and including this packet, moving into position on the first picture B of this first set of pictures of the second program,
starting with this transport packet TP corresponding to a bidirectional-image B beginning and including this packet, skipping the header of the PES packet and then replacing the data of the payload of each transport packet of the video component with the data of the replacement pictures,
after all the data of the replacement picture have been inserted, replacing video stuffing such as numbers 0 for the data of the payload of the transport packet TP of the component with video stuffing such as xe2x80x9c0xe2x80x9d figures until the next transport packet TP of this second-program video component of which the PUSI indicator is set at 1, excluding this transport packet TP,
thereupon, if the number of bidirectional pictures preceding the said first picture is larger than 1, repeating the two above stages for the ensuing bidirectional pictures B, that is beginning with the next transport packet of which the PUSI indicator is set at 1.
Said pictures of each of said sets of pictures constituting the video components of the first and second programs also may be moved by transport packet streams, each transport packet being fitted with a PUSI indicator which when set at 1 denotes that said packet contains the beginning of a PES packet containing a single picture, but where the second stream lacking random access indices RAI. Said method of the invention is then characterized by consisting in:
after the switch command, determining the first transport packet TP of the second-program video component which is present in the stream transport and fitted with a PUSI indicator set at 1 and of which payload data begin with a video sequence header, where called for followed by a group of pictures GOP header,
searching for the sequence header either, if present, immediately following the PES packet header or, if the video component is scrambled, by unscrambling,
if the ascertained TP packet comprises an adaptation field AF with program clock reference PCR and in the case that the component carries the program clock, setting the discontinuity indicator in this transport packet TP,
starting from this transport packet TP (included), move into position on the first bidirectional picture B of this first set of pictures of the second program,
starting form this transport packet TP corresponding to a beginning of a picture B included, move past the PES packet or replace this PES header situated at the beginning of the payload with a reconstructed PES packet header and then replacing the payload data of each transport packet TP of the video component with the data of the bidirectional replacement picture B, and if the transport packets TP are scrambled, denoting them in clear by setting the transport scrambling control TSC at the binary value 00,
after all picture replacement data have been inserted, replacing the data of the payload if the ensuing transport packets TP of the component with video stuffing such as 00 octets while continuing in denoting them in clear until the next transport packet TP of this second-program video component of which the PUSI indicator is set at 1 (this transport packet TP excluded),
then if the number of B pictures preceding the first picture P is larger than 1, repeating the two above stages for the following B pictures, that is starting with the next transport packet TP of which the PUSI indicator is set at 1.
In another feature of the invention, the inserted transport packet TP exhibits the following features:
the PUSI indicator is set at 0,
the status of the continuity counter is set at that of the preceding transport packet TP continuity counter of the video component,
the adaptation field control AFC is set at the binary value 11 denoting that an adaptation field AF and a payload are present in this transport packet TP, the discontinuity indicator situated in the AF shall be set at 1
if the video component carries the program clock, move a program clock reference PCR computed according to the preceding PCR(s) of the same component into the AF.
In one embodiment variation including pre-processing to allow setting the RAI""s provides the following features of the inserted transport packet TP:
the PUSI indicator is set at 1,
the status of the continuity counter is set at that of the preceding transport packet TP continuity counter of the video component,
the adaptation field control AFC is set at the binary value 11 denoting that an adaptation
the random access indicator RAI situated in the adaptation field AF is set at 1,
if the video component carries the program clock, a program clock reference PCR calculated from the preceding PCR of the same component shall be moved into the adaptation field AF,
the PES packet header that was in the replaced transport packet TP is moved into the payload of this inserted transport packet TP and in case the presentation stamp PTS is absent from the PES packet header, said PTS shall be computed and moved into the PES packet header data.
In another feature of the invention, the reconstructed PES packet header assumes the following form: start code=0x000001; stream ID=that of the first packet; PES packet length=0x0000, 0x000001; flag octet=0x8500; length of the header data=0x00; PES packet in clear; no PTS/DTS nor any other PES packet header data.
In another feature of the invention, if the ascertained transport packet TP does not contain an adaptation field AF or if it involves the component including the program clock and the adaptation field AF lacks a program clock reference PCR, said method consists in replacing the particular transport packet TP by a transport packet TP bearing a program clock reference PCR and in shifting the replaced transport packet TP as well as the following transport packets TP of this video component in the transport stream until one of them can be inserted into a stuffing transport packet TP.
In another feature of the invention, the switching procedure consists in switching at the beginning of the first set of pictures of the second program encountered after the switch command, which may take place at any picture of the current set of pictures of the first program. In order to circumvent regulation problems relating to the decoder""s buffer memory, the method of the invention advantageously replaces the transport packets containing each picture of the second program""s first set(s) of pictures encountered after the switch command with an equal number of transport packets containing a supercompressed corresponding picture and stuffing data.
This method can be implemented when a main program (for instance a nationwide program) is decoupled toward a secondary program (for instance a local one).
In another feature of the invention, the switching procedure consists in interrupting the first program at the beginning of a predictive picture following the switch command, in inserting a predetermined-content picture pattern, then in switching at the beginning of the first set of pictures of the second program encountered after the switch command. As above, in order to circumvent regulation problems attending the decoder buffer memory, the method of the invention advantageously consists in replacing the transport packets containing each picture of the last set(s) of pictures of the first program encountered after the switch command with an equal number of transport packets containing a supercompressed corresponding picture and stuffing data.
This procedure may be implemented when reconnecting a secondary program such as a local program onto a main program such as nationwide one.